Motion estimation is an effective inter-frame coding technique to exploit temporal redundancy in video sequences. Motion-compensated inter-frame coding has been widely used in various international video coding standards The motion estimation adopted in various coding standards is often a block-based technique, where motion information such as coding mode and motion vector is determined for each macroblock or similar block configuration. In addition, intra-coding is also adaptively applied, where the picture is processed without reference to any other picture. The inter-predicted or intra-predicted residues are usually further processed by transformation, quantization, and entropy coding to generate compressed video bitstream. During the encoding process, coding artifacts are introduced, particularly in the quantization process. In order to alleviate the coding artifacts, additional processing has been applied to reconstructed video to enhance picture quality in newer coding systems. The additional processing is often configured in an in-loop operation so that the encoder and decoder may derive the same reference pictures to achieve improved system performance.
FIG. 1A illustrates an exemplary system block diagram for an video encoder based on High Efficiency Vide Coding (HEVC) using adaptive Inter/Intra prediction. For Inter-prediction, Motion Estimation (ME)/Motion Compensation (MC) 112 is used to provide prediction data based on video data from other picture or pictures. Switch 114 selects Intra Prediction 110 or Inter-prediction data and the selected prediction data is supplied to Adder 116 to form prediction errors, also called residues. The prediction error is then processed by Transformation (T) 118 followed by Quantization (Q) 120. The transformed and quantized residues are then coded by Entropy Encoder 122 to form a video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packed with side information such as motion, mode, and other information associated with the image area. The side information may also be subject to entropy coding to reduce required bandwidth. Accordingly, the data associated with the side information are provided to Entropy Encoder 122 as shown in FIG. 1A. When an Inter-prediction mode is used, a reference picture or pictures have to be reconstructed at the encoder end as well. Consequently, the transformed and quantized residues are processed by Inverse Quantization (IQ) 124 and Inverse Transformation (IT) 126 to recover the residues. The residues are then added back to prediction data 136 at Reconstruction (REC) 128 to reconstruct video data. The reconstructed video data may be stored in Reference Picture Buffer 134 and used for prediction of other frames.
As shown in FIG. 1A, incoming video data undergoes a series of processing in the encoding system. The reconstructed video data from REC 128 may be subject to various impairments due to a series of processing. Accordingly, various in-loop processing is applied to the reconstructed video data before the reconstructed video data are stored in the Reference Picture Buffer 134 in order to improve video quality. In the High Efficiency Video Coding (HEVC) standard being developed, Deblocking Filter (DF) 130 and Sample Adaptive Offset (SAO) 131 have been developed to enhance picture quality. The in-loop filter information may have to be incorporated in the bitstream so that a decoder can properly recover the required information. Therefore, in-loop filter information from SAO is provided to Entropy Encoder 122 for incorporation into the bitstream. In FIG. 1A, DF 130 is applied to the reconstructed video first and SAO 131 is then applied to DF-processed video.
A corresponding decoder for the encoder of FIG. 1A is shown in FIG. 1B. The video bitstream is decoded by Video Decoder 142 to recover the transformed and quantized residues, SAO/ALF information and other system information. At the decoder side, only Motion Compensation (MC) 113 is performed instead of ME/MC. The decoding process is similar to the reconstruction loop at the encoder side. The recovered transformed and quantized residues, SAO/ALF information and other system information are used to reconstruct the video data. The reconstructed video is further processed by DF 130 and SAO 131 to produce the final enhanced decoded video.
In the High Efficiency Video Coding (HEVC) system, the fixed-size macroblock of H.264/AVC is replaced by a flexible block, named coding unit (CU). Pixels in the CU share the same coding parameters to improve coding efficiency. A CU may begin with a largest CU (LCU, also referred as CTU, coded tree unit in HEVC). The initial coding unit is then partitioned using quadtree. Usually, a coding system uses Rate-Distortion Optimization (RDO) to determine whether a CU should be further split. After a coding unit is split by the quadtree, the resulting coding units may be further split using quadtree unless the coding unit reaches a pre-specified smallest CU (SCU) size. The collection of quadtree partitions of a picture to form variable-size coding units constitutes a partition map for the encoder to process the input image accordingly. The partition map is conveyed to the decoder so that the decoding process can be performed accordingly. In HEVC, the CTU size for the luma component (i.e., Y)—may be up to 64×64. The CU size is often referred to as 2N×2N. When the CU is split by the quadtree, it results in four N×N CUs.
In addition to the concept of coding unit, the concept of prediction unit (PU) is also introduced in HEVC. Once the splitting of CU hierarchical tree is done, each leaf CU is further split into prediction units (PUs) according to prediction type and PU partition. The Inter/Intra prediction process in HEVC is applied to the PU basis. For each 2N×2N leaf CU, a partition size is selected to partition the CU. A 2N×2N PU may be partitioned into 2N×2N, 2N×N, or N×2N PU when Inter mode is selected. When a 2N×2N PU is Intra coded, the PU may be partitioned into either one 2N×2N or four N×N.
In the HEVC standard, a coding profile, named Main Profile has been finalized and the Main Profile only supports color video in the YUV420 chroma subsampling format. Color video comprises multiple color components, and the color components in the form of luminance and chrominance are usually used in the field of video coding. Among various color component formats, YUV or YCrCb is often used for video coding. The YUV chroma subsampling format applies full sampling to the luma component (i.e., Y) and 2:1 horizontal subsampling and 2:1 vertical subsampling to the chroma component (i.e., Cr or Cb). FIG. 2 illustrates examples of sampling patterns associated with YUV420, YUV422 and YUV444 chroma sub-sampling patterns. An area of color samples (4 pixels×2 rows) is shown in block 210 for YUV420 chroma sub-sampling. The corresponding full sampling for the Y component is shown in block 211. For each chroma component (i.e. Cr or Cb), only two samples are formed, as indicated by shaded areas 1 and 2 in block 212 for the area of 4×2 color pixels. While the YUV420 format is widely used for general consumer applications, in some pro-consumer and professional applications, higher fidelity in colors is desirable and the YUV422 and YUV444 formats are used. For YUV422 chroma sub-sampling of an area of 4×2 color pixels 220, the sampling for the Y component 221 is the same as before. However, for the chroma component, the sub-sampling forms 2×2 sub-sampled pattern 222. For the YUV444 chroma sub-sampling of an area of 4×2 color pixels 230, the sampling for the Y component 231 is the same as before. However, for the chroma component, a full sampling (i.e., 1:1 sub-sampling) is used to form 4×2 sub-sampled pattern 232. The sampling locations for the chroma components may be aligned with the luma samples. Nevertheless, the sampling locations for the chroma components may also be offset from the luma samples horizontally, vertically or both horizontally and vertically.
In HEVC Main Profile, quadtree split is applied to transform blocks, where the transform process is applied to each coding unit (CU) and uses the CU size as the initial transform unit (TU) size. Each TU can be partitioned by quadtree. FIG. 3 illustrates an example of quadtree partition of a TU. Block 310 corresponds to quadtree partition of a largest coding tree block (CTB) where each square corresponds to a leaf CU. Transform process is applied to the residues associated with a CU indicated by thick-lined block 312. The CTB can be 64×64, 32×32, 16×16 or 8×8. The initial TU 320 corresponds to the residues of a selected CU (i.e., block 312). The initial TU 320 is partitioned by quadtree to form next level TUs 330. TUs 330 can be further partitioned by quadtree to form TUs 340. The transform size for HEVC can be 32×32, 16×16, 8×8 or 4×4. When the YUV422 or YUV444 format is used, video coding tools and schemes for all these chroma sub-sampling formats will be needed. For example, the H.264/MPEG-4 AVC (Advanced Video Coding) standard includes a profile named Fidelity Range Extensions (FRExt), which enables higher quality video coding by supporting increased pixel bit depth and higher-resolution color formats, such as YUV422 and YUV444 chroma sub-sampling. It is desirable to develop coding tools such as structure and partitioning method for coding unit, prediction unit and transform unit associated with YUV422 and YUV444 chroma sub-sampling formats of the HEVC coding system.